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Desert Research and Technology Studies (Desert RATS) is a multi-year series of 
hardware and operations tests carried out annually in the high desert of Arizona on 
the San Francisco Volcanic Field. These activities are designed to exercise planetary 
surface hardware and operations in conditions where long-distance, multi-day roving is 
achievable, and they allow NASA to evaluate different mission concepts and approaches 
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in an environment less costly and more forgiving than space. The results from the RATS 
tests allow selection of potential operational approaches to planetary surface explora- 
tion prior to making commitments to specific flight and mission hardware develop- 
ment. in previous RATS operations, the Science Support Room has operated largely in an 
advisory role, an approach that was driven by the need to provide a loose science 
mission framework that would underpin the engineering tests. However, the extensive 
nature of the traverse operations for 2010 expanded the role of the science operations 
and tested specific operational approaches. Science mission operations approaches from 
the Apollo and Mars-Phoenix missions were merged to become the baseline for this 
test. Six days of traverse operations were conducted during each week of the 2-week 
test, with three traverse days each week conducted with voice and data communica- 
tions continuously available, and three traverse days conducted with only two 1-hour 
communications periods per day. Within this framework, the team evaluated integrated 
science operations management using real-time, tactical science operations to oversee 
daily crew activities, and strategic level evaluations of science data and daily traverse 
results during a post-traverse planning shift. During continuous communications, both 
tactical and strategic teams were employed. On days when communications were 
reduced to only two communications periods per day, only a strategic team was 
employed. The Science Operations Team found that, if communications are good and 
down-linking of science data is ensured, high quality science returns is possible 
regardless of communications. What is absent from reduced communications is the 
scientific interaction between the crew on the planet and the scientists on the ground. 
These scientific interactions were a critical part of the science process and significantly 
improved mission science return over reduced communications conditions. The test also 
showed that the quality of science return is not measurable by simple numerical quantities 
but is, in fact, based on strongly non-quantifiable factors, such as the interactions between 
the crew and the Science Operations Teams. Although the metric evaluation data suggested 
some trends, there was not sufficient granularity in the data or specificity in the metrics to 
allow those trends to be understood on numerical data alone. 

Published by Elsevier Ltd. 


1. Introduction — overview of the Desert RATS 2010 
operation 

The 2010 Desert RATS test was a 14-day operation 
conducted at Black Point in the San Francisco Volcanic 
Field near Flagstaff, AZ, with 2 small-pressurized rover 
prototypes. Each rover was operated with a 2-person crew 
consisting of an engineer/commander and a scientist, 
testing a variety of operations approaches within a 
scenario of human exploration of the lunar surface (see 
[1] for a complete description of 2010 and preceding 
years’ RATS missions). Each rover crew conducted a 
7-day mission with a mid-mission crew change at the 
end of Day 7. Mission operations oversight was provided 
by a Mission Operations Team from the Mission Opera- 
tions Directorate at Johnson Space Center, which operated 
out of a remote Mission Control Center (rMCC) located at 
Black Point Base Camp, and a variety of science operations 
teams also operating at Black Point and in Flagstaff. 

The overarching goal of the test was to operate the 
rovers under contrasting communications states and modes 
of rover operations (see Table 1). In order to understand the 
effects of different communications conditions on science 
operations, three traverse days each week were conducted 
with complete voice and data communications available 
throughout each 24-h day with no delay between transmis- 
sion and reception (called “Continuous Comm”, or CC). The 
remaining three traverse days simulated operations with 
two % 1-h orbital communications satellite passes per day, 
a condition referred to as twice-a-day comms, or 2/day, that 


might be in extant during exploration on the lunar far side. 
In this test case, the assumption was that communications 
took place every 12 h. In addition to varying communica- 
tions conditions, rovers were operated either as a mutually 
supporting team (called “Lead and Follow”), with each rover 
simultaneously exploring the same terrain, or with each 
rover in communications contact, but working in different 
portions of the field area (called “Divide and Conquer”). The 
testing and communication schedule is delineated in 
Table 1. In addition to the 12 days of active exploration, 2 
days were devoted to operations in a Habitat Demonstration 
Unit (HDU) Pressurized Excursion Module (PEM) (see [2]). 

The Desert RATS 2010 test employed a variety of 
science operations teams whose task was to manage 
science operations from support rooms either at the Black 
Point Base Camp or in Flagstaff, AZ. Selection of the team 
members who would man the Science Support Rooms 
was a critical decision in that the quality of team would 
relate directly to the quality of the results. Each team 
selected was composed of scientists with a background in 
planetary sciences, and who were at a mix of career levels 
ranging from graduate students to senior scientists. This 
selection ensured that each team member would have a 
background in the science operations conducted by the 
rover crews and would be better prepared to evaluate the 
mission. The Science Operations Team consisted of 38 
members drawn from 5 NASA Centers, 2 International 
Partner organizations, 8 universities, and 3 government 
agencies, and had over 400 years of accumulated experience 
as scientists working all over the world, including over 37 
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Table 1 

Desert RATS testing schedule. 


Sunday, 29 
Aug 

Monday, 30 
Aug 

Tuesday, 31 
Aug 

Wednesday, 1 
Sept 

Thursday, 2 
Sept 

Friday, 3 
Sept 

Saturday, 4 
Sept 

Pre-test 

stand-down 

Dry run 

Test Begins; 
CC w/Lead & 
Follow Ops 

CC w/Lead & 
Follow Ops 

CC w/Lead & 
Follow Ops 

2/Day; Divide & 
Conquer Ops 

2/Day; Divide & 
Conquer Ops 

Sunday, 5 Sept 

Monday, 
6 Sept 

Tuesday, 7 Sept 

Wednesday, 
8 Sept 

Thursday, 9 Sept 

Friday, 10 Sept 

Saturday, 11 Sept 

2/Day; Divide & 
Conquer Ops 

PEM Ops; CC 

Rover crew change out; 
CC w/Lead & Follow Ops 

CC w/Lead & 
Follow Ops 

CC w/Lead & 
Follow Ops 

2/Day; Divide & 
Conquer Ops 

2/Day; Divide & 
Conquer Ops 

Sunday, 12 Sept 

Monday, 13 Sept 

Tuesday, 14 Sept 

Wednesday, 
15 Sept 

Thursday, 
15 Sept 

Friday, 16 Sept 

Saturday, 17 Sept 

2/Day; Divide & 
Conquer Ops 

PEM Ops; CC 

Test ends 






years of combined geologic fieldwork experience. In addi- 
tion, professional diversity was equally critical, given the 
variety of scientific and operational problems encountered. 
Members of the RATS Science Operations Team had techni- 
cal backgrounds in geology, planetary science, astronomy, 
mining engineering, spaceflight operations, mechanical 
engineering, and experience in such diverse areas as field 
geology, oil and gas exploration, geophysics, information 
technology, mining engineering, mine automation and 
robotics, mineral economics, oceanography, Space Shuttle 
image analysis, International Space Station research facility 
development, aerospace engineering, planetary science mis- 
sion operations, geochemistry, volcanology, impact crater- 
ing, igneous petrology, tectonics, and lunar sample analysis. 
Lastly, experience on space missions was also extensive, 
with team members having participated in 5 Apollo mis- 
sions, 2 Mars Exploration Rover missions, 27 Space Shuttle 
Missions, 6 International Space Station expeditions, NASA- 
Mir, Stardust, Long-Duration Exposure Facility, Mars Recon- 
naissance Orbiter, Lunar CRater Observation and Sensing 
Satellite, Mars-Phoenix, MAVEN, Fermi Gamma Ray Tele- 
scope and the BioSAR mission. 

Each Science Operations Team was responsible for testing 
science operations management concepts in concert with the 
larger mission control team, conducting either real-time, 
tactical science operations during the day or strategic plan- 
ning operations conducted at night. Each scientist rotated 
through several assignments as part of the test. The purpose 
of this rotation was: (1) to provide exposure to different 
ground support functions within each team, thus developing 
a cadre of experienced science support personnel for future 
NASA missions; and (2) to evaluate the quality of the science 
operations throughout the test. In particular, exposure to 
multiple aspects of the operation by each individual was 
expected to provide a more statistically balanced evaluation 
of the quality of the operation, and helped eliminate single- 
point biases that might be introduced with a more limited 
exposure by each team member. 

The purpose of this paper is to document the lessons 
learned by the Science Operations Team during the course of 
Desert RATS 2010, in order to apply those lessons to future 


Desert RATS tests. In addition to the experience of the Science 
Operations Team, a critical task of the overall RATS 2010 test 
was to also document the lessons learned from of the rover 
crews during the conduct of the test. Several papers in this 
volume, including Love and Bleacher [3]; Bleacher et al. [4]; 
and, Hurtado et al. [5], summarize the crew experience and 
the lessons learned by the geologists and astronauts who 
were serving in the rovers on Desert RATS 2010. Lastly, Bell 
et al. [6], summarize the lessons learned by the Mission 
Operations Directorate crew in the rMCC. 

2. Science operations management — historical 
perspectives from crewed and robotic science missions, 
with applications to the 2010 RATS test 

Management of science activities during real-time 
mission operations has been critical to improving the 
science return of NASA missions. The Apollo human lunar 
expeditions and the Mars-Phoenix mission provided con- 
trasting approaches to the science operations manage- 
ment during planetary surface exploration. 

2.1. Apollo lunar missions 

Although the early Apollo missions were pursued as 
largely engineering exercises, later Apollo missions con- 
centrated on extensive scientific exploration of the lunar 
surface. Understanding the nature of surface mission 
operations during the Apollo missions requires assess- 
ment of both the nature of the individual parts and how 
they were successfully woven together to produce the 
very synergistic scientific legacy of Apollo Lunar Explora- 
tion Program. Years prior to the Apollo 1 1 landing, it was 
fully appreciated by science leaders and NASA manage- 
ment that President Kennedy’s challenge to the Nation 
represented an unprecedented opportunity for global 
scientific leadership and establishing a scientific legacy 
that would forever change the way in which we under- 
stand the origin and evolution of the Earth and planets. 
This overarching understanding led to the development of 
significant scientific input to all aspects of the missions, 
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ranging from landing site selection; planning of lunar 
surface operations and traverses; selection and funding of 
a wide array of scientific instrumentation to be utilized 
and deployed both in orbit and on the surface; astronaut 
selection and training, including astronaut geologic field 
and classroom training (e.g., [7]); development of new 
hardware to assist and extend scientific operations (e.g., 
the Lunar Roving Vehicle); development of new opera- 
tions techniques to optimize and extend exploration (e.g., 
Standup EVAs from the LM cabin); field operations train- 
ing that included astronauts, scientists, mission opera- 
tions staff and NASA managers working as a team during 
joint integrated mission simulations in the field; and 
extensive post-mission debriefings and lessons-learned 
discussions (see, e.g., Compton [8] or Baldwin [9] for a 
comprehensive discussion of science planning and execu- 
tion during Apollo). All of these background activities 
contributed significantly to successful surface operations 
on the Moon, but perhaps the most fundamental lesson 
from the Apollo experience is that optimal planetary surface 
operations in the future must be based on broad, synergistic 
foundation of science, astronauts and operations. 

The actual landing on the Moon and onset of surface 
operations was preceded a wide range of activities that 
developed the partnerships that worked so well during 
mission operations. Starting soon after their selection, 
astronauts received geological training in the classroom 
and on field trips (see, e.g., [7]). When an astronaut was 
chosen for a particular mission, they participated in the 
discussions concerning the selection of the landings sites 
and worked with scientists to familiarize themselves with 
the choices involved, the scientific goals and objectives 
and the potential operational challenges (H. Schmitt and 
J. Head, personal communication, 2009 and 2012). Once 
sites were selected, astronauts, operations personnel and 
scientists worked closely together to develop lunar sur- 
face exploration traverse plans which were then simu- 
lated and tested in field training trips with the prime and 
backup astronaut crews, the astronauts who would talk 
with the flight crew on the lunar surface (CAPCOMs), 
scientists, operations personnel, and mission controllers. 
These types of close coordination activities developed 
partnerships, respect, mutual insights and understanding. 
Scientists understood the complexity of operations and 
the time constraints of being on the lunar surface, astro- 
nauts understood the scientific objectives and how they 
could accomplish and exceed them, mission operations 
personnel understood the importance of the scientific 
objectives and developed insights and new techniques 
into how they could assist in their accomplishment, and 
flight control personnel learned how to organize and 
implement the surface operations in order to optimize 
the ability to accomplish and exceed the scientific objec- 
tives for the mission. Many of these types of simulations, 
both in the field and in Mission Control in Houston took 
place before launch. In addition, astronauts, CAPCOMs, 
traverse planners and scientists had many briefings in the 
months prior to launch in Houston and at the Kennedy 
Space Center on details of the specific mission traverses 
and mission operations. At these meetings, the specific 
geology of the landing sites, the scientific objectives of the 


mission, and the implementation of these objectives 
through a set of pre-planned traverses and sampling 
operations were reviewed and discussed (J. Head, F. Horz, 
personal communication, 2012), and inputs from the 
astronauts were sought and implemented. 

In the conduct of each mission, the purpose of the flight 
control team that operated out of the Mission Operations 
Control Room (MOCR) at Johnson Space Center was to 
manage the time of the crewmembers on the lunar surface 
in order to meet mission timelines, and evaluate the 
health and status of the crew and spacecraft in real time, 
in order to be able to respond to emergencies as soon 
as possible [13]. During lunar surface operations, the 
CAPCOMs were scientist-astronauts who had worked 
extensively with the crew during training and were highly 
versed in the mission science objectives (W. Phinney, 
F. Horz and G. Lofgren, personal communication, 2009 
and 2010). Hierarchically below the MOCR positions were 
a wide variety of back room positions that directly 
supported the primary flight controllers in the MOCR, 
including a Field Geology Experiment Support Room. In 
the case of science support on Apollos 15-17, this Support 
Room was in turn assisted by an informal group of scientists 
in a different room who could follow the operations in real 
time, but who had limited capability to influence operations 
taking place in the chain of support rooms above them. The 
most important purpose of these science support teams was 
to provide the front room controllers with science expertise 
that, in conjunction with the expertise in science operations 
developed by the scientist/astronaut/flight control com- 
bined team, would make real-time decisions on science 
operations questions raised on the lunar surface (J. Head, 
personal communication, 2012). 

Each mission developed, prior to launch, a detailed 
science support plan in order to ensure that mission 
science objectives were met (e.g., [10-12]). The purpose 
of these plans were to, "... describe the manner in which 
the Flight Operations Directorate (FOD) plans to execute 
the mission control functions for the experiments an 
integrally related operational objectives assigned... in 
the Mission Requirements Document [13].” In addition, 
a stated objective of these plans was to, “attempt to 
nullify the effects of events perturbing the mission plan 
and recover nominal mission plan execution conditions 
[13],” and, “...[to] maximize the capability to move into 
alternate modes of operation when quick-look data 
review indicates unusual opportunities exist to maximize 
science objectives [13].” The pre-planned traverses and 
individual activities at the stations were designed as a 
“nominal plan”, but were in no way thought of as a “pre- 
determined script”. The broad scientific objectives were 
always kept in mind by the astronauts and the scientists, 
and it was understood that if new developments occurred 
during surface operations that could accomplish the 
objective better than the preplanned activities, than these 
new objectives clearly had priority and that it was the call 
of astronaut on the surface as to what to do. For example, 
the discovery by Dave Scott and Jim Irwin of the “Genesis 
Rock” (Apollo sample number 15415), the “green glass 
beads”, and the “Seat Belt basalt” on Apollo 15 clearly called 
deviation from the nominal plan. Due to the extensive and 
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close pre-mission training and coordination, scientists on 
the ground had great confidence in the astronauts on the 
surface to recognize the critical new discoveries and to act 
accordingly (J. Head, personal communication, 2012). In a 
similar manner, when operations on the surface deviated 
from the nominal plan (e.g., time expended extracting the 
drill on Apollo 15), scientists, engineers and operations 
personnel on the surface revised the nominal plan while 
the astronauts on the surface continued their focused 
operations. 

By the time a given Apollo mission was executed, a 
well-integrated team had developed. Astronauts on the 
surface conducted scientific analysis and exploration. 
Mission operations staff and mission controllers main- 
tained seamless operations and monitored hardware 
behavior in order to optimize the time that the astronauts 
on the surface had for science and exploration. Scientists 
monitored the progress of the astronauts on the surface 
and assessed their scientific descriptions and sample 
acquisition progress and if necessary, the Science Back 
Room in the Mission Control Center could provide broad 
input to the crew through the CAPCOM. However, the 
scientists were confident in the ability of the crew to 
identify and respond correctly to new discoveries. The 
general attitude was, “Let the astronauts do the job they 
are trained to do and be ready to help when necessary.” 
Post-EVA debriefs provided the opportunity to give 
updates and revised traverses, but the exploration of the 
surface was undertaken by the astronauts, representing 
the combined Science/Mission Operations Team. 

This team effort was the key to mission success in the 
Apollo Lunar Exploration Program and resulted in its 
enduring scientific legacy. The lessons learned from this 
endeavor are: (1 ) involve all team members together early 
in the process and build synergism, (2) work together in 
the scientific training of the astronaut partners so that 
they can operate on the surface as explorers, not puppets, 
and (3) stand ready to help at any time during their 
independent surface operations and exploration, and 
expect to exchange input between periods of surface 
activity. Clearly, during longer periods of surface opera- 
tions on future planetary expeditions, the long time scale 
of communication will necessitate a significant degree of 
astronaut independence and autonomy [14]. Thus the 
Apollo experience may serve as an essential and helpful 
guide for developing and optimizing this synergism. 

2.1.3. Mars-Phoenix mission 

The Mars-Phoenix Mission was a robotic lander mis- 
sion that operated on the Martian surface between May 
and November 2008 [15]. Although not a rover mission, 
the Mars-Phoenix Mission Team built its operations 
approach on lessons learned in the Mars Exploration 
Rover mission [16]. A critical factor in Mars-Phoenix 
mission operations was managing data return from the 
surface within a limited return data budget, which drove 
the Mars-Phoenix team to operate essentially two plan- 
ning processes: (1) a tactical process that dealt exclu- 
sively with those science activities whose commands 
were up-linked to the spacecraft each Martian day, and 
(2) a strategic process that considered those science 


activities that would be executed over the longer planning 
horizon of the total mission duration [16]. The require- 
ment for teams to operate under different planning 
horizons was a function of mission duration. In particular, 
both the MER and Mars-Phoenix missions operated for 
much longer periods of time than the Apollo missions, 
and consequently had a much longer time to accomplish 
mission science operations. 

The tactical science operations planning process on 
Mars-Phoenix determined the specific science data that 
could be collected within a single Martian day, which was 
balanced against the spacecraft capabilities, engineering 
constraints and the daily data up- and down-link limita- 
tions of the spacecraft and the Deep Space Network. To 
that end, a given tactical planning process involved active 
interplay between engineering constraints and science 
objectives. Using these constraints and objectives, the 
tactical planning process generated a series of commands 
that could be uploaded to the Mars-Phoenix lander in 
time for execution in the upcoming day (see [16], Table 2, 
for an excellent graphic on the Mars-Phoenix tactical 
process). 

Bass and Talley [16] defined strategic processes as 
including, “...the generation of midterm and also long- 
term ‘strategic' science plan that describes the possible 
lander actions from 4 to 11 sols in the future.” Conse- 
quently, the strategic science operations process for Mars- 
Phoenix involved an analysis of high-level science objec- 
tives that would be decomposed into more detailed 
activities. These detailed activities would, in turn, feed 
the tactical planning process for each day. It is expected 
that the 2-way communications travel times between 
targets outside of cis-Lunar space will, in the future, 
necessitate conducting a strategic science process with 
human landed missions in a manner similar to Mars- 
Phoenix, and the 2010 RATS Science Operations Test 
was designed, in part, to test that process with human 
exploration crews. 

2.1.2. Application of the Apollo and Mars-Phoenix science 
Operations processes to the desert RATS 2010 science 
Operations test 

During pre-mission planning, it became clear that the 
communications design planned for the mission would 
allow conversations between Mission Operations Teams 
at Black Point Base Camp and the crew in the field to 
occur without any communications delay. This operation 
was largely identical to kind of operations carried out on 
Apollo, which were characterized as “tactical” in nature. 
Given the length of the planned 2010 mission, it was also 
clear that assimilating and analyzing the data produced 
during each day’s operations would be an important 
process for influencing plans on subsequent traverse days. 
Consequently, the science team decided that in addition 
to tactical science operations on RATS 2010, the Science 
Operations Team would conduct a strategic science 
operation in order to test the efficacy of strategic planning 
on the conduct of future human planetary surface opera- 
tions. This test would integrate operations approaches 
proven to be successful on both human and robotic 
planetary exploration programs. 
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Table 2 

Science Data Collection Metric. The individual scores from each sub-area were summed to give the total score. Data 
collected on Desert RATS 2010 are shown graphically in Figs. 4 through 6. 


Rating Definition 


Description 


Regional context 


0 

None 

No geologic context described 

1 

Limited 

Sketchy geologic context described 

2 

Sufficient 

Geologic context described and understood in Science Support Room; 
in particular, how do the units at this site relate to the units in previous sites 

Stratigraphy 

0 

None 

No stratigraphic relations described 

1 

Limited 

Relationship of stratigraphic units not described 

2 

Sufficient 

Stratigraphic relationships between units described and information 
understood in the Science Support Room 

Stratigraphic sampling 


0 

None 

No stratigraphic units sampled 

1 

Limited 

Not all stratigraphic units sampled 

2 

Sufficient 

All stratigraphic units sampled 

Sample description 


0 

None 

No sample descriptions 

1 

Limited 

Not all samples were adequately described 

2 

Sufficient 

Samples were sufficiently described and understood 
in the Science Support Room 

Sample collection 


0 

None 

No Sample numbers and collection boxes described 

1 

Limited 

Not all sample numbers and collection boxes described 

2 

Sufficient 

Samples numbers and collection box designation were sufficiently 
described and understood by the Science Support Room 

Summary 

0-10 

Combined score 

A maximum score of 10 would mean that all science available science data 
from a given station had been acquired during the activities at that station 


3. RATS 2010 science Operations test overview 

The objectives for science operations at Desert RATS 
2010 were to (1) evaluate operations with two Tactical 
Science Operations Teams (TSOT), and a Strategic Science 
Operations Team (SSOT); and (2) evaluate a methodology 
for measuring science productivity as a function of varying 
communications conditions. The TSOTs were responsible 
for managing real-time operations of the crewmembers at 
each station and assumed operational control of the 
crewmembers from the Test Director and the Mission 
Operations Test Team whenever they were off the rovers 
and conducting exploratory science. During the traverses 
between stations, the Test Director and the Mission 
Operations Directorate control team managed the rovers, 
although all visible images and verbal communications 
with the crew were available to each TSOT in real time. 
The SSOT was responsible for evaluating the scientific 
output of each day’s activities and conducting, as neces- 
sary, replanning efforts after each day’s traverse opera- 
tions had been completed. Traverses would be replanned 
based on either science objectives missed or serendipitous 
science discoveries that potentially would alter mission 
science objectives. In addition to the Strategic and Tactical 
Science Operations Teams, a Field Science Operations 
Team (FSOT) operated during each traverse day. The 
FSOT’s job was to evaluate crew performance without 
the filter of the communications net that existed for the 
other teams. In particular, the FSOT members looked at 
crew performance on the various geologic tasks as a way 


Operations 
Management 
Flow 


Mission 

Management 

Team 


Strategic Science 
Operations Team 


« 

l 


Tactical Science 
Operations Team 
Rover A 

“V 


< 1 — 


Tactical Science 
Operations Team 
Rover B 

v 


« 

4 

ft 


/ . j 




Fig. 1. Science team relationships on Desert RATS 2010. The job of the 
Field Science Operations Team (FSOT) was to operate as a loss-of- 
communications backup, and under nominal operations would not have 
an input into the information flow depicted in this chart. In the event of 
a loss of communications, the FSOT would assume the responsibilities of 
the Tactical Science Operations Teams. 


of understanding, where necessary, how communications 
affected the remote Science Operations Team s’ perception 
of science effectiveness. In addition, it was planned that 
the FSOT would take operational control of the science 
mission in the field if a major breakdown in communica- 
tions took place. However, no serious communications 
failures occurred during the 2010 RATS mission and this 
function was not tested. 
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As part of the overall science mission management 
activity, a designated Mission Lead Scientist participated 
daily in Mission Management Team (MMT) meetings (see 
Fig. 1 below), where science results and their potential 
impact on the overall operation were discussed with the 
Mission Manager and other team leads, including the 
Rover Team, the Mission Operations Team and the Astro- 
naut Office representative. The Science Team participated 
as a full member of the Mission Management Team, and 
science objectives, as much as engineering objectives, 
drove decision making throughout the test. In addition 
to MMT participation, the Mission Lead Scientist 
debriefed the rover crews at the end of the day, gave a 
“hand-off’ briefing to the SSOT each evening at the start 
of their shift, and conducted the morning science briefing 
to the rover crews. 

3.1. Science operations team schedules 

The TSOT operated from 7:00 AM to 5:30 PM each day of 
continuous communications and PEM operations (see 
Fig. 2). Each rover had a dedicated TSOT. During 2/Day 
communications test, the TSOTs stood down from opera- 
tions. Each member of the team was responsible for a 
particular segment of the operation, including overall man- 
agement of daily science objectives, interfacing with the 
Mission Operations Team, management of sample docu- 
mentation, collecting data on geologic context, and evaluat- 
ing data from a variety of cameras and science instruments 
on the Rovers. The two TSOTs operated out of dedicated 
Science Support Rooms at Black Point Base Camp. 

The SSOT shift began at roughly 8:00 PM, ending when 
a data analysis and replanning effort was complete. A 
typical SSOT shift ended at «- 4:00 AM. Each individual on 
the SSOT was responsible for analyzing a particular 
science data set and developing an understanding of what 
was accomplished during the previous day, recommend- 
ing on the basis of that analysis if any changes needed to 
be made to the following day’s activities. 

4. Detailed overview of RATS science team operation 

4.1. Tactical Science Operations Team (TSOT) 

The TSOT had three objectives: (1) evaluate the effec- 
tiveness of science data collection during each shift; (2) 


provide technical assistance to the rover crews during 
geologic operations; and, (3) evaluate the effectiveness of 
the science operations setup as implemented for Desert 
RATS 2010. Each tactical team worked with an assigned 
rover crew, coordinating with the Test Director to accom- 
plish the daily science plan. In addition, the Science Leads 
for each TSOT conducted face-to-face tag-ups while rovers 
were traversing between stations to resolve any issues that 
arose at a given station. 

4.1.1. TSOT membership 

Each TSOT was staffed with the following team mem- 
bers, who operated console positions in each TSOT: 

• Team Lead (SCIENCE) — Responsible for all science 
operations decisions for a given rover. 

• Science Communicator (SC1COM) — Direct voice link 
with the rover crew; patterned after the CAPCOM 
position in the higher level test control room, SCICOM 
was the only Science Support Room person to interact 
directly with the crew during EVA operations. 

• Documentarian — Responsible for maintaining a diary 
of all science operations, including problems that arose 
and notable science accomplishments. 

• Operations Team Liaison (OPSL1NK) (co-located with 
Test Director) — Responsible for communication “hand- 
offs" between the Test Director (TD) and the Science 
Team Lead as well as acting as liaison between teams 
on any operational issues. 

• EV1 Science Lead (EV1 ) and EV2 Science Lead (EV2) — Re- 
sponsible for overseeing the science operations of the 
EV1 crewmember, including data captures from EVA 
system cameras (see Hurtado et al. [5] and Bleacher et 
al. [4], for specific descriptions of camera assets available 
to the crew), as well as keeping track of science descrip- 
tions while on EVA and assisting the Science Team Lead 
with science questions arose during EVAs. 

• GigaPan Operator (GIGAPAN) — Responsible for initiat- 
ing data acquisition, download and initial evaluation of 
image quality from the GigaPan camera system on 
each rover, and providing an evaluation of the geology 
of the local rover area to assist the Science Team Lead 
with real-time science questions during EVA. 

• MastCam Operator (MASTCAM) — Responsible for initi- 
ating data acquisition, download and initial assessment 
of image quality from the MastCam camera system on 
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Fig. 2. Daily operations schedule for RATS 2010. 
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each rover, and providing an evaluation of the geology 
of the local rover area to assist the Science Team Lead 
with real-time science questions during EVA. 

When a rover crew was working GeoLab operations 
during PEM operations days, the appropriate TSOT room 
was used to manage these operations (see Evans et al. [2], 
for a detailed description of GeoLab design and opera- 
tions). Data acquisition activities were conducted from 
the TSOT room using the GEOIMAGE and GEODATA 
system (see Evans et al. [2]) using available consoles that 
would otherwise be used for rover operations. 


4.1.2. TSOT evaluation of science operations 

The TSOTs maintained track of the daily timeline as the 
mission proceeded, ensuring that the following activities 
occurred: 

• science objectives were met for each mission phase; 

• sample documentation data was collected in real time; 

• geologic context information was provided by the 
crewmembers; 

• timelines for each station were executed as planned. 

Each TSOT member was responsible for evaluating the 
effectiveness of science activities on RATS 2010 using a 
daily science score sheet (see Tables 2-5). These forms 
were designed to evaluate whether the activities per- 
formed at a given location were effective at determining 
geologic context and geologic sample character, evaluat- 
ing sample documentation procedure, and to help deter- 
mine the causes of errors when science data collection 
was less than optimal. 


4.2. Strategic Science Operations Team (SSOT) 

4.2.1. Team objectives 

The Strategic Science Operations Team (SSOT) objec- 
tives were to (1) assimilate and synthesize the data 
collected in the day’s activities to determine if planned 
objectives had been met, or if there were any discoveries 
in the previous day’s traverses that would necessitate 
replanning efforts for the following day’s traverses; 
(2) conduct traverse replanning; (3) set priorities for 
downlinked data products for the upcoming tactical 
process; (4) identify and prioritize samples for further 
analysis on GeoLab operations days; and, (5) evaluate the 
quality of science data derived in the field by the rover 
crews and filtered by the Tactical Science Operations Team. 

4.2.2. Team methodology 

The SSOT operated in a phased approach: the imaging 
teams and the individual Rover Science Data Teams 
reviewed their data sets and prepared recommendations 
for SSOT Lead and the rest of the team. If replanning of the 
following day’s activities was called for, the rest of the 
SSOT participated in this effort in order to be completed 
by the following morning’s uplink briefing. See Fig. 3 for a 
typical strategic planning process. 

4.2.3. SSOT membership 

The SSOT was staffed by 10 geoscientists who per- 
formed the following functions. Some teams were staffed 
with multiple individuals, who shifted job assignments as 
the strategic evaluation and planning process proceeded 
through a given shift. 

• Team Lead — Responsible for management and comple- 
tion of all activities of the SSOT, including replanning of 


Table 3 

Science Operations Metric. As with the Science Data Collection Metric, the individual scores from each sub-area were summed to give the total score. 


Area rated 

Rating 

Descriptor 

Operations process 

0 

Unacceptable 


1 

Borderline 


2 

Acceptable 

Operational roles 

0 

Unacceptable 


1 

Borderline 


2 

Acceptable 

Science data/ 

0 

Unacceptable 

hypotheses tracking 

1 

Borderline 


2 

Acceptable 

Operational 

0 

Unacceptable 

leadership 

1 

Borderline 


2 

Acceptable 

Operations facilities 

0 

Unacceptable 


1 

Borderline 


2 

Acceptable 

Summary 

10 

Maximum 


score 


Definition 


Process does not allow science team to address the posed science questions 

Process is working but improvements are warranted 

Process allows science team to address the posed science questions 

Role not required for successful operations, i.e., duplication of efforts among roles. 

Role required, but improvements are warranted 

Role required and no improvements necessary (i.e., work load adequate, etc.) 

Process does not make sufficient data available to evaluate a given hypothesis 
Process makes available some data, but many questions remain after a given operation 
Process allows sufficient data to clearly evaluate hypotheses 

Leadership inadequate to allow operations team to complete the science mission 

Leadership provides some assistance to the operations team in completion of the science mission, but 

team is often unable to complete science mission due to lack of direction 

Leadership provides assistance and direction during science operations process; makes it easy for the 
team to complete the science mission 

Operations team is unable to acquire and process data, and manage the science mission 
Operations team is able to acquire and process data, and manage the science mission, but with 
significant difficulty 

Operations team is able to acquire and process data, and manage the science mission with no 
difficulty 

Science operations worked flawlessly and no improvements necessary 
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Table 4 

Science Merit Metric. 


Rating Descriptor Definition 


1 

2 

3 

4 

5 

6 


Limited 

Adequate 

Sufficient 

Significant 

Exceptional 

Discovery 


Data does not allow understanding of the scientific context of the field area 
Data reaffirms existing hypotheses and facts 

Data reaffirms existing hypotheses and facts in new areas or levels of detail 
Data potentially elucidates or modifies existing hypotheses in new areas or level of detail 
Data potentially resolves a major scientific question or highly significant hypothesis 
Data potentially introduces a novel idea or hypothesis 


Table 5 

Technology/Operations Implementation Metric. 


0 Poor 

1 Limited 

2 Good 

3 Significant 

4 Exceptional 


Complete failure of instrumentation or total operator error led to missing or largely incomplete geologic data and/or sample 
documentation information. 

Instrumentation failure or major operator error led to minimal geologic context and sample data available, but not sufficient to 
establish a large scale understanding of the geologic terrane or sample character. 

Hardware functioned well and operator technique captured sufficient information to establish large-scale terrane context and 
outcrop distribution, and established acceptable sample documentation data. 

Operator technique captured significant information that enabled critical information capture at the outcrop level as well as 
establishing broader contextual framework and established acceptable sample documentation data. 

Operator technique captured information that enabled basic research to be done from the captured images and data without the 
augmentation of separate sample research. In addition, operator collected acceptable sample documentation data. 


traverses for the following day’s science operations and 
development of the daily science plan. 

• Documentarian — Responsible for keeping a real-time 
log of all activities that occurred within a given shift. 

• Strategic Operations Lead — Responsible for managing 
long-term operational constraints that affected the 
daily replanning process, including communications 
constraints, long-term mission objectives, and con- 
sumables constraints. 

• Activity Planners — Responsible for preparing the 
revised daily plan for each rover crew based on the 
recommendations of the SSOT Lead. 

• Long-Term Planning Lead — Responsible for (1) coordi- 
nating science team groups working on datasets that 
were critical to planning the next day’s tactical activ- 
ities; (2) determining whether there were any critical 
discoveries that warranted further study; (3) deter- 
mining whether there were any issues that would pose 
a continuing threat to achievement of mission science 
objectives; (4) revising the traverse plans; and, (5) 
leading the end-of-day science discussion. 

• Geology Team — Responsible for analyzing the data set 
produced by the Rover imaging and science teams, and 
making specific recommendations to the Strategic 
Operations Lead on revising the following day’s geo- 
logic traverses. 

• Mineralogy/Petrology Team — Responsible for analyz- 
ing the data set produced by the Rover imaging and 
science teams, making specific recommendations to 
the Strategic Operations Lead on revising the following 
day’s geologic traverses, maintaining sample prioriti- 
zation, and identifying key samples for further analysis 
on PEM days. 

• Rover Imagery Data Team — Responsible for assimilat- 
ing the data sets from each rover team from a given 
day and preparing a summary of results, including 


(1) key results; and, (2) specific key images that illu- 
strated operations during the day's traverse activities. 

• GigaPan Data Team — Responsible for reviewing the pre- 
vious days GigaPan imagery data and preparing a sum- 
mary for the SSOT of ( 1 ) the images from each traverse 
station; and, (2) specific key images that illustrated the 
accomplishments at a given traverse station. 

• Rover Science Data Team — Responsible for assimilating 
specific geologic data to prepare a summary for the SSOT 
of that day’s science accomplishments, including (1) 
selecting key images and science results for presentation 
to the traverse planners; (2) assessing completion or 
non-completion of day’s plan; (3) identifying “broke 
points” or areas of unusual accomplishments that might 
affect the following day’s plan; and, (4) keeping a log of 
all samples collected, including sample description, cri- 
tical curation data, and an assessment of what samples 
warrant further studying in the PEM. 


4.2.4. Schedule 

SSOT daily operations commenced after the end of the 
day’s traverse operations, when the data from the imaging 
systems and voice communications were downloaded and 
delivered to the Embassy Suites Conference Facility. Initial 
assimilation of data took a 4 h. At the end of the assim- 
ilation phase, the SSOT Lead determined whether the 
following day’s activities needed to be modified. When 
modification was called for, the team prepared a revised 
traverse plan that would be briefed to the Test Director 
prior to the following morning’s uplink briefing to the 
rover crew (see Figs. 2 and 3). The objectives of critical 
team meetings are described below. 

4.2.4. 1. SSOT Kick-off Meeting. During the SSOT Daily Kick- 
off Meeting, the SSOT Lead provided a brief overview of 
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comprehensive plan for next 
day’s science objectives 
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engineering constraints to 
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Assimilates data from the 
day’s traverses 

• “Cherry picks” key 
images, science results 

Determines completion or 
non-completion of day’s plan 

• Identifies “broke points” 
or areas of unusual 
accomplishments 


Fig. 3. Strategic Science Operations Team process. 


(1) the present current mission success criteria status 
and planned objectives for the day just completed; (2) 
reviewed and prioritized downlinked data sets expected; 
(3) presented resource estimates for the following day’s 
planned activities (communication schedule, data volumes 
[critical and non-critical], and total rover operation time; (4) 
presented any mission updates from MMT; (5) outlined the 
meeting schedule for shift activities. 

4.2.42. Science Assessment Team hand-off to Traverse 
Planning Team. The Science Assessment Team consisted 
of the Rover Imagery Data Team (see Fig. 3), the GigaPan 
Data Team and the Rover Science Data Teams. Their job 
was to assimilate data from the previous day’s traverse 
and prepare a detailed summary of the day’s operations. 
Once the Science Assessment Team completed the review 


and collation of the previous day’s downlinked data, it 
handed off data products to the Traverse Planning Team 
to be considered by a combined SSOT Team meeting. This 
meeting discussed (1) key science data from the day’s 
activities; (2) assessment of completion status of each 
science objective from the day’s activities; and, (3) 
identification of key events, including either issues that 
prevented completion of planned science objectives or 
key, unexpected science results that might merit further 
study on the following day. 

4.2.43. Science Activity Plan Approval Meeting. At the 
completion of each Traverse Planning Team activity, the 
Long-Term Planning Lead presented the proposed traverse 
plan for the next day to the SSOT. This meeting was run as a 
free and open discussion of the proposed plan for the next 
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day’s science activities. At the end of the meeting, the SSOT 
Lead either approved the plan or indicated areas where 
further replanning efforts were needed prior to presentation 
to the MMT. 

4.2.5. SSOT evaluation of science operations 

As with the TSOTs, each SSOT member was responsible 
for evaluating the effectiveness of science activities on 
RATS 2010 using an identical daily science score sheet. 
During continuous communications days, SSOT evalua- 
tions added to the data set provided by the TSOT, with the 
different perspective of making the evaluations on the 
basis of recorded data that was several hours old. The 
SSOT was concerned with determining: 

• If science objectives were met for each mission phase 

• The quality of the downlinked datasets 

• If sample documentation data was collected 

• If geologic context information was provided by the 
crewmembers 

4.3. Field Science Operations Team (FSOT) 

The Field Science Operations Team was responsible for 
operating in the field with the rover crews, assessing the 
operation at each field locality that a particular Rover 
crew investigated. 

4.3 A. Team objectives 

The objective of the FSOT was to provide on-site 
evaluations of the crews’ observations and sample selec- 
tions relative to the actual geology present at any given 
site. They monitored compliance with existing sampling 
protocols and associated photo documentation, and also 
judged the crew’s performance during communication 
outages to evaluate the effectiveness of the Tactical Team 
in supporting the field activities. 

4.3.2. Team methodology 

Each FSOT consisted of a 2-person team that evaluated 
how each crewmember conducted their EVA science 
operations. Each team accompanied a rover during all 
traverse science operations. The emphasis was on under- 
standing, at the “receiver end”, how both communica- 
tions and operations approaches affected the science 
being done in the field. In particular, it was critical for 
each science team evaluator to decide the degree to which 
the science data collection conducted in the field was 
being affected, either in a positive or negative sense, by 
the communications or operational scenario in effect. The 
FSOT did not interact with the crew in the field, but acted 
as the Science Operations Team ’s eyes and ears on the 
ground, particularly in situations where communications 
were compromised. In addition, the FSOT was tasked with 
assuming the management of science operations in the 
field if a significant loss of communication between the 
crew and the TSOT, caused by an unexpected event such 
as weather or major equipment malfunction, prevented 
continuation of the nominal mission for periods in excess 
of minutes. In this role, the field team kept track of 


timelines and the progress of operations, particularly 
during traverse stops, to fill in the role that would 
normally be undertaken by the SCICOM at a particular 
traverse station. 

5. Evaluation of Desert RATS 2010 science operations 

A critical aspect of the Desert RATS 2010 test was 
evaluating the quality of the science return during the 
mission as a function of variable communications condi- 
tions. This evaluation was considered a high priority by 
both the Science Operations Team and the overall mission 
management. It was hoped that a clear understanding of 
the factors that affect the quality of the science return 
would assist in major architectural decisions for future 
planetary surface missions. 

In order to make this evaluation, the Science Opera- 
tions Team attempted to use parameters that were either 
measurable or could be used unambiguously to evaluate 
the achievement of a particular goal. To that end, a series 
of science operations evaluation sheets were prepared 
and filled out by science team members on each Science 
Operations Team (TSOT, SSOT and FSOT) daily at the 
completion of a particular operations shift. These evalua- 
tion sheets are shown in Tables 2 through 5. In addition to 
the numerical metrics, each evaluation included written 
comments on the particular parameter being measured, 
whether it related to geologic data or the overall opera- 
tions set up. Evaluations involving fatigue, workload and 
working conditions were also completed at the beginning, 
middle and end of a shift. 

5.1. Evaluating science merit — consideration of numerical 
metrics design and data collection 

As NASA budgets have come under increasing pres- 
sure, the effort to understand the effectiveness of different 
planetary exploration approaches has led to the collecting 
of numerical metrics on a variety of test parameters. This 
effort is reasonable and laudable; in particular, unambig- 
uous, properly collected numerical test data can help 
make decisions, save development time, and settle differ- 
ences between devotees of conflicting approaches. A 
problem arises, however, when numerical metrics are 
collected on activities that are subjective and non-numer- 
ical, such as measuring science productivity or scientific 
efficiency. Failure to understand both the mathematical 
basis of a particular evaluation and the legitimate math- 
ematical operations can be carried out on a particular data 
set can lead to misunderstanding of the importance of a 
given activity or, worse, lead to incorrect programmatic 
decisions made on the basis of poorly understood test 
results. 

In many cases, it is valid to use numerical values to 
evaluate a particular science task. When conducting 
statistical analyses of a particular parameter, such as the 
radiometric ages of a given rock unit, more measurements 
of the sample’s age allows definition of the accuracy of the 
resultant measurement. However, metrics collected by 
assigning numerical designators to a given characteristic 
have an underlying assumption that the assigned value is, 
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in fact, a number that has an absolute quantitative value 
and can be used in the statistical manipulation of study 
results. Problems arise if this assumption is incorrect. 
Stevens [17], Lord [18] and Gass [19] have discussed the 
pitfalls of conducting numerical rankings without having 
a clear understanding of the numerical character of the 
qualities being ranked. 

Stevens [17] distinguished between cases where (1) a 
number is a title, such as a designator of a player in a 
sport (Nominal Scale); (2) rank ordering of different 
objects or events using a parameter that can establish 
order but does not define a well-ordered, regular numer- 
ical difference, such as the results of a horse race (Ordinal 
Scale); (3) scales where there is a uniform interval 
between individual members of a set, but where the set 
interval is not defined in an absolute sense, such as the 
Celsius temperature scale (Interval Scale); and, (4) scales 
where the measurement is a ratio between an absolute 
zero quantity and a unit scale length, such as the Kelvin 
temperature scale (Ratio Scale). The critical consideration 
in all these scales is that while each represents numerical 
data, statistical manipulation of the data must be based 
on the kind of data collected. Statistical manipulation of 
nominal or ordinal data beyond finding the mode, med- 
ian, percentile and Chi-square, is meaningless. Conse- 
quently, it is important to understand the type of 
numerical scale used for metric data collected in tests 
such as Desert RATS before statistical manipulations of 
the resulting data set are carried out and conclusions 
reached as to the quality or efficacy of a particular 
approach. Ultimately, numerical data can be collected on 
many aspects of scientific research in a given discipline, 
but the measurement approaches must be fully consistent 
with the discipline being studied. Failure to understand 
the nature of a given scientific discipline can lead to the 
collection of metric data that is flawed, and can, in turn, 
miss what is important scientifically and what is not. 

5.2. Desert RATS science metrics collection 

The Science Operations Team collected metric data on 
four separate areas: Science Data Collection (Table 2), 
Science Operations (Table 3), Science Merit (Table 4) and 
Technology/Operations Implementation (Table 5). The 
measurements collected were the result of long discus- 
sions within the Desert RATS Team about the specifics of 
metrics collection, and at the start of the operation, it was 
felt that the designed metrics had a reasonable chance to 
successfully discriminate differences between test states. 
In particular, it was hoped that these metrics would help 
settle a long-standing debate about whether crews on a 
planetary body would conduct better science operations 
under continuous two-way communications or with 
reduced 2/Day communications conditions. 

Ultimately, the results of the numerical data collection 
were disappointing in that the metric data alone rarely 
indicated more than general trends, and did not provide 
sufficient detail to allow the Science Operations Team 
to understand the underlying factors that contributed to 
those trends. Although analysis is on going, the authors feel 
that the design of the numerical metrics was inadvertently 


flawed for a variety of reasons (the raw metric data is 
available from the Corresponding Author on request). The 
discussion below of one metric illustrates the problems we 
have encountered in post-test analysis of the data. 

5.2.3. Science Data Collection Metric data example: 
preliminary results and limitations 

The Science Data Collection Metric was used to rate 
the quality of the geological data collected by each 
crewmember when conducting traverses in the form 
of either observations of geologic context or samples 
(Table 2). This metric required team members from both 
the TSOT, the SSOT and the FSOT to score a variety of 
activities carried out by crew during the geological 
characterization of each field station (see Horz (2012, this 
volume), for the specifics on traverse design). Further, a 
total score for Science Data Collection Metric was created 
by summing the scores of each of the five individual 
activity areas. The TSOT conducted the ratings in real- 
time after the crew completed their station tasks, while 
the SSOT rated their performance each night after a 
review of the down-linked voice, video and electronic 
still image data. 

The data set for the Science Data Collection Metric is 
plotted as a function of geological field station in Figs. 4 
through 6. Prior to the start of Desert RATS 2010, it was 
expected that interaction between the crew and the TSOT 
during continuous communications would eliminate, or 
at least greatly reduce, mistakes or omissions, resulting in 
generally high scores for this metric. It was further 
expected that high TSOT scores would be mirrored by 
similar scores from the SSOT. During 2/Day communica- 
tions, the TSOT would not be in a position to correct these 
kinds of procedural mistakes, and it was hypothesized 
that the metric scores would be lower for the SSOT, given 
that crew operations would have not any real-time 
Science Operations Team oversight. 

The data sets presented in Figs. 4 through 6 illustrate the 
difficulty the science team encountered in post-mission data 
analysis, and two issues became clear during post-test data 
compilation. First, individual scores clustered toward the 
highest score (see Fig. 4), with only limited values < 2. This 
lack of granularity between highest and lowest scores made 
it impossible to draw any meaningful conclusions about the 
quality of the individual operations conducted. In particular, 
it was impossible, on the basis of the raw individual activity 
scores, to determine if there were any flaws in the field 
operation that could be corrected in future field tests. 

The second issue is that the combined metric scores 
plotted as a function of field location for both teams 
indicates a general trend toward higher scores from the 
TSOT (Fig. 5) on days with continuous communications as 
compared with the SSOT scores (Fig. 6) on the same days. 
Given that operations with continuous communications 
should have provided the highest quality science data for 
the SSOT to analyze, high SSOT scores would be expected 
during continuous communications. This was not observed. 
Further, SSOT scores during 2/day communication days 
when the TSOT was not operating were generally higher, 
and the implying better science quality than SSOT scores 
during continuous communications days. In comparison, 
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Fig. 4. Strategic Science Operations Team Science Data Collection Metric data for individual sub-categories. The data consists of 65 data points collected 
throughout the Desert RATS 2010 test (see Table 2 for definition of evaluation criteria). Note the significant overlapping of data at the highest score. 



Fig. 5. Strategic Science Operations Team Science Data Collection Metric data combined score. The data consists of 65 data points collected throughout 
the Desert RATS 2010 test (see Table 2 for definition of evaluation criteria). Blue boxes denote periods of continuous communications. 


metric data collected by the Field Science Operations Teams 
during the whole test indicate a generally constant level of 
performance of the crew in the field, regardless of commu- 
nications state (Fig. 6). 

The overall data set implies that the variation in metric 
scores between the TSOT and SSOT were influenced by 
factors other than communications state. In short, the 
hypothesis that science data quality would be controlled 
solely by communications state was found to be not valid, 
given the consistency of metric scores from the FSOT. It is 
likely that some component of the interaction between the 
TSOT and the crew introduced a bias in the data set that 
affected the SSOT scores, but the pure numerical metrics do 
not provide insight into the root cause of the difference in 


scores. Written comments by Science Operations Team 
members, discussed below, provides better insight into the 
factors affecting science team operations but so far, no 
single factor has been identified that directly affects the 
quality of science return on Desert RATS 2010. Written 
comments by SSOT members indicated that the volume of 
data to be analyzed on any given traverse day was extre- 
mely large, cumbersome to assimilate and of a highly 
variable nature, making it difficult to adequately analyze 
all the data during a given 8-h shift (see Section 6.6). It is not 
clear how any element of the TSOT process could have 
affected this condition and led to the variation in metric 
scores. Additional written comments by SSOT members also 
indicate that the practice of placing team members into 
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Fig. 6. Tactical Science and Field Science Operations Team Science Data Collection Metric data combined score. The data consists of 146 data points collected 
throughout the Desert RATS 2010 test (see Table 2 for definition of evaluation criteria). Blue boxes denote periods of continuous communications. During 2/day 
communications periods, the only tactical assets operating were the Field Science Operations Team. 


many different jobs during the course of the 2-week test 
may have introduced significant inefficiency into the data 
analysis process, but again, it is unclear how this would 
affect the metric scores. Consequently, it is possible that 
these scores are related less to crew performance or com- 
munications state and more to lack of job experience in the 
Science Operations Team members on any given day. 

Although the data set from the Science Data Collection 
Metric provided the most obvious example of this dis- 
connect between numerical scores and test conditions, 
the team experienced similar problems with other metric 
data sets. A critical lesson learned in the process of 
collecting science metrics is that numerical data alone 
do not allow an understanding of the root causes of 
variations in metric scores. 

6. Overall lessons learned 

6.1. Team composition and qualifications 

The professional experience of the Science Operations 
Team is critical. In particular, the team cannot be made up 
exclusively of either non-scientists or junior scientists with 
little experience. As delineated above, the RATS Science 
Operations Team was extremely diverse in experience, age 
and science background. Having an experience level con- 
sistent with the mission goals applies to the rover crews as 
well. Desert RATS 2010 was the first such integrated test 
where science crewmembers were specifically selected for 
their abilities and breadth of experience as field geologists. 1 


1 In the early years of the Desert RATS series, test subjects were 
selected for geologic field experience, but those tests were devoted 
solely to 1-2 h duration pressure garment tests and not multi-day, 
integrated tests. See Ross et al. [1] for more details. 


In addition, the astronauts that completed each rover team 
had, as a minimum, specific training in sampling protocols, 
and the astronaut team included one individual with 
advanced degrees in planetary science. Consequently, each 
rover team had a breadth of science training that exceeded 
previous years’ crews. 

6.2. Communications state 

6.2A. Continuous communications 

In continuous communications with stable, high fidelity 
voice and image data, the quality of the overall science is 
better than communications states where communications 
are intermittent or have long intervals between contacts. 
This is a function of the ability of the science team to 
interact with the crewmembers on the outcrop, in real time, 
and science return was greatly improved by that interaction, 
which was not available on 2-a-day communications. The 
nature of this interaction was primarily the scientific 
discussions that were held between the rover crew in the 
field and the TSOT supporting them. The Science Operations 
Team member supporting one of the crewmembers on EVA 
noted, “When all systems were working properly, we got 
really good data (TSOT A, EVA Science Controller (ES) 2, 
Mission Day 1, Technology and Operator Performance 
Metric Database)”. Another science team member noted, 
“The EVA process worked very well. Again, an educated 
crew following proper protocol provided sufficient data for 
science to determine overall regional context and general 
site geology. EVA Crewmember 1 (EV1 ) and EVA Crewmem- 
ber 2 (EV2) have become veiy adept at communicating with 
SC1COM and each other without voices running over each 
other (TSOT B, Science Communicator (SCICOM), Flight Day 
2, Science Operations Metric Database).” As it was during 
the Apollo Program, the crewmembers on the planet’s 
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surface are members of a larger science team, and team 
interaction in science always improves the quality of the 
final product. However, communications will not always be 
stable, but a diligent TSOT and a well-trained, scientifically 
competent astronaut crew can develop methods to work 
around ratty or intermittent communications, particularly 
when bad communications are expected. It was noted, 
“Limited communications today during first EVA led to 
decision to brief crew prior to and following their EVA. 
The outcome appears to be veiy good, and demonstrates 
that science objectives can still be met under such con- 
strained operations ... requires well-trained crew and 
Science Operations team. Involving science team members 
in operational roles allows team members to fully appreci- 
ate operational constraints. [This is] better than having 
science team as just advisors/observers of the ops. (TSOT 
B, ES2, Mission Day 3, Technology and Operator Perfor- 
mance Metric Database).” 

With bad communications during the continuous com- 
munications phase, science return was limited and led to a 
loss of critical science data, such as sample documentation 
and geologic context. It was noted that, “[The] process is 
working when comm is working. We are unable to evaluate 
the ability of the crew to address all questions without 
adequate communication for much of the day... Questions 
remain about details of outcrop and sample descriptions 
due to comm limitations. (TSOT B, Documentarian, Mission 
Day 9, Science Operations Metric Database).” 

6.2.2. 2/Day communications 

During 2/Day communications state, science analysis 
and return was directly related to how well the crew 
gathered the image data and provided the verbal context 
for that data. Poor imaging system deployment, missed or 
poorly aligned backpack photographs, and poor or missing 
crew field notes or missed science objectives resulted in 
degraded science return. In short, 2/Day communications 
state requires crewmembers to do the appropriate imaging 
data runs correctly each time, as the SSOT will not see the 
final image products until after it is no longer possible to re- 
occupy a given station. One comment noted, “Unclear 
context — crew never noted if they were sampling bfl and 
b!2 or just one or the other (Strategic Team, Long Term 
Planner, Mission Day 4, Science Merit Metric Database).” 

However, when the rover crew and the systems all 
worked well, the science data collected was as good as 
what was achieved during continuous communications. 
For example, one SSOT member noted, “Crew members 
did a great job of capturing stills at their sites. EV2’s 
final thoughts at this stop with the HiDEF video imagery 
were particularly helpful for context (Strategic Team, 
Geological Data Evaluator (GeoData), Mission Day 4, 
Technology and Operator Performance Metric Database).” 

6.3. Operations team interactions 

The TSOTs undertook detailed scientific discussions in 
real time to test science hypotheses, and to decide 
subsequent courses of action by the crew to improve the 
science return from the field. These science discussions 
were a critical part of the TSOT process and mission 
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science, and were independent of communications delay. 
They were critical to improving science return, and have 
formed an important part of every human and robotic 
mission NASA has ever flown. In particular, one TSOT 
member noted, “When we had communications today we 
were able to formulate hypotheses in real time along with 
the crew and provide guidance to them prior to EVAs to 
look for specific evidence or specific samples to test those 
hypotheses (TSOT A, ES2, Mission Day 2, Science Opera- 
tions Metric Database)." 

The TSOTs, the SSOT and the Mission Operations 
Directorate team were able to mesh together as a team 
and work well together in spite of limited pre-mission 
time for mutual training and familiarization. This speaks 
well for the commitment of all members of the team to 
accomplish the science tasks in spite of limited training 
and opportunities to work together prior to assembling 
in Arizona. It was also a critical component of mission 
operations, as the Science Operations Team on any mis- 
sion is only one facet of the overall mission. In particular, 
if the Science Operations Team cannot work effectively 
with the Mission Operations Team, then effective science 
return becomes much more difficult. It was noted that, 
“The ops folks did a pretty good job of keeping us 
informed when there were issues which allowed us to 
make decisions and re-plan in real time to get the best 
science... They were responsive when we informed them 
of issues (TSOT A, ES2, Mission Day 1, Science Operations 
Metric Database)." 

One area that was not managed well, largely because 
its value was not understood prior to the RATS 2010 
exercise, was the need for hand-off briefings between all 
elements of the mission management structure on both 
2/day and continuous communication days. Hand-off 
debriefs between the TSOT leads and the SSOT lead at 
the beginning of a given shift during continuous commu- 
nication days ensures all science personnel have the same 
information on critical “on-the-fly” lessons learned and 
daily science accomplishments. During 2/Day communi- 
cations days, it was equally critical to conduct quality 
briefs of the crew in the morning and debriefs of the crew 
in the afternoon. This ensures that the crew understands 
science lessons learned from the previous day and the 
critical activities in the upcoming day, as well as making 
sure that the SSOT understands, at the end of a particular 
day, the science results that stood out as well as the 
lessons learned and science accomplishments that should 
receive particular attention during the subsequent SSOT 
session. 

6.4. Science team leadership 

Good science operations team leadership is crucial for 
success. If the leadership is good, it can help smooth out 
problems and make decisions that help the team mem- 
bers proceed, even in the absence of fully functioning 
systems. Both the TSOTs and the SSOT benefited from 
good leadership throughout the test. In particular, the 
team leads made sure that each team member understood 
the daily mission, and what was expected of them 
throughout the day. It was noted that, “The tasks of the 
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science ops team were clearly outlined at the beginning of 
the session, together with the goals of the session itself. 
Timeliness of the work was also stressed and reiterated 
throughout (SSOT, ESI, Mission Day 5, Science Operations 
Metric Database).” 

6.5. Numerical metrics on science quality 

As discussed in Section 5, the science metrics collected 
on Desert RATS 2010 did not provide unambiguous 
answers to the architectural questions this test hoped to 
address. This may largely be a problem in test design, 
which was undertaken by science team members versed 
in the science operations planning but having no profes- 
sional expertise in designing and implementing these 
kinds of evaluations. It argues strongly that the numerical 
metric data collection process must be based on measure- 
ment systems designed by professionals in operations 
research, working in conjunction with the scientists who 
understand the science operations to be conducted. 

A larger issue with the collection of numerical metrics is 
that science return is more than the statistics of boots-on- 
the-ground time, samples collected or sites occupied. 
Science return is the understanding of the geology of a 
given planet achieved through interaction between the 
scientists on the crew and the scientists on the ground, 
and this is not easily measured on the basis of numerical 
data alone. Without an understanding of the critical inter- 
action between scientists in the process of collecting data 
and testing ideas, pure numerical data will be unable to 
provide a clear understanding of the reason for a particular 
numerical score. 

6.6. Training and changes in work assignments 

Early on in the plan for Desert RATS 2010, it was 
decided that science team members would be shifted 
among a number of job assignments every 3 days. The 
rationale behind this was both the desire to get as much 
training for the individual scientists, and to increase the 
number of people making inputs to the metrics database 
from a given position. In particular, it was hoped that 
switching jobs would eliminate any bias within the metric 
database and improve the effectiveness of the data 
collection. However, the practice of moving people to 
new jobs every 3 days meant that no one was in a position 
long enough to know their jobs well. Consequently, 
metric data may not represent a “steady state” condition, 
but in fact may only be quantifying team members’ 
inexperience. One team member noted, “...training has 
been largely trial by fire, which has its limitations (TSOT 
A, Mast Camera Data Evaluator (MastCam), Mission Day 9, 
Science Operations Metric Database).” 

6.7. Data analysis activities in support of strategic science 
operations 

The enormous volume of data coming out of any given 
day of this mission was difficult to review, assimilate, and 
evaluate the data by the SSOT in the planned 8-h shift. 
This was partly a problem of data volume, and partly a 


problem of inefficient manipulation and mining of data 
sets. In particular, the tools available to the SSOT mem- 
bers were insufficient to allow the SSOT to evaluate the 
data volumes produced during daily operations, and 
metrics collected on SSOT operations were subsequently 
flawed. One SSOT member noted, “1 spent over 4 h trying 
to translate the video/audio and consequently was not 
ready for the assessment meeting. We could really use a 
software program that translates the audio into a typed 
format for our review; it would allow hours more time for 
interpretation of data (SSOT, GeoData, Flight Day 5, 
Technology and Operator Performance Metric Database).” 
In particular, the trends in the Science Data Collection 
Metric discussed in Section 5 may be related to inefficient 
and frustrating data mining and analysis techniques 
imposed on the SSOT. Two possible ideas may be consid- 
ered: first, when comparing SSOT to TSOT operations, the 
real-time assessment, integration and discussion conducted 
by the TSOTs during the course of a day may have given the 
TSOT a better understanding of the geology than was 
possible for the SSOT to achieve during an 8-h shift. 
Alternatively, the higher metric scores recorded by the SSOT 
during 2/Day communications may represent decreases in 
SSOT frustration when analyzing data sets unencumbered 
by the additional TSOT data that was added to the databases 
during continuous communications conditions. 

One additional consideration with regards to science 
data analysis during the mission lies in the issue of what 
was really needed. The SSOT based its task on an initial 
assessment of all the data that was available, which led to 
the problems detailed above. Part of the future pre- 
mission planning should be to consider, in consult 
between the science planners and the Mission Operations 
Team, what data will be needed to make the kinds of real- 
time operations decisions, and to design the data systems 
to make that data easier to extract. 

7. Suggestions for improvement 

Desert RATS 2010 was a successful test, in that it 
integrated science operations concepts from both human 
and robotic planetary surface missions. However, there were 
important issues uncovered during the conduct of the test 
that should be addressed prior to conducting similar opera- 
tions in the future. The suggestions to improve future 
operations gleaned from this test can be summarized as 
follows: 

• If collection of numerical metrics is going to be an 
activity in this kind of test, individuals with an opera- 
tional research background must be engaged to help 
design better metrics than those employed this year. 

• Qualitative comments by team members will be as 
critical to improving science operations as the quanti- 
tative metrics. In particular, the ability to effectively 
and consistently capture qualitative evaluations must 
be a critical part of the test evaluation protocol. 

• Krikalev et al. [14] have raised the issue of “creeping 
determinism” in ISS science operations, particularly 
with respect to reduction in crew autonomy and it’s 
affect on the research being conducted on that 
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platform. Although there was no effort to evaluate this 
issue on RATS 2010, an understanding of the benefit of 
different levels of autonomy should be undertaken. 
Further, it will be critical to differentiate between 
creative science interaction between the crew on a 
planet’s surface and a ground science team, and more 
restrictive “over the shoulder” supervision of in-space 
crew by ground controllers. 

• If the evaluation of continuous communications on 
science operations is a critical test objective, the com- 
munications infrastructure should ensure that high-fide- 
lity, stable communications exists at all science stations. 

• Team members need to stay in a particular job greater 
than 3 day at a time. 

• Team members need sufficient pre-mission training 
and practice exercises to ensure that a team member’s 
experience in the tests is a reflection of what is work- 
ing well and what is not, rather than being a reflection 
of inadequate training. This is also critical for the 
crewmembers conducting the field geologic investiga- 
tions on the ground. 

• Improved technology is needed to make it easier and 
faster to assimilate data sets. For instance, it is simple 
to speed read transcripts, but impossible to "speed 
listen” to a sound file at high playback rates. 

• The data set that the SSOT needs to assimilate within a 
given shift should be better defined in order to 
increase operational efficiency. 

• Data mining needs to be facilitated with consistent, 
clear file structures and graphical user interfaces 
so reviewing a particular data set does not lead to a 
lengthy “treasure hunt”. 

• Better capability for manipulation of datasets, includ- 
ing GIS-based interfaces, is needed to extract to critical 
data from large mission data sets for SSOT operations. 

• The crew needs to be able to review imaging data sets in 
real time to correct any imaging deficiencies prior to 
leaving a particular locality, and this needs to be part of 
the nominal list of tasks completed by the crew during 
science operations. This applies to all communications 
conditions, but it is clearly a requirement for reduced 
communications where a Science Support Team cannot 
review the quality of a down linked image and correct 
deficiencies in real time. Improving the capability of the 
crew to manage the quality of the data they are collecting 
will improve mission science return and crew autonomy, 
and allow science operations teams to be more engaged 
in scientific discovery rather than data collection over- 
sight and crew time management. 

The planning for Desert RATS 201 1 has been underway 
during the preparation of this manuscript, and the lessons 
learned in 2010 are already being applied to the next test 
in the series. 
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